The preset invention relates to a moving object detection apparatus and method, and is preferably applied to a moving object detection device in a video surveillance system, for example.
A video surveillance system equipped with a moving object detection device has been installed in buildings, warehouses and the like for crime prevention. This moving object detection device processes image signals supplied from surveillance cameras to detect a moving object in an image corresponding to the image signals.
It is desirable that the moving object detection device can accurately detect moving objects in images even if the device is affected by the following factors: diffused reflection light caused by water waves; natural disturbance such as plants shaken by the wind; diffused reflection light caused by flags flying in the strong wind; diffused reflection light caused by headlights in the night; and disturbance caused by artificial materials, such as a sudden change of brightness by turning on/off lights.
To satisfy the requirement, a moving object detection device is proposed which detects a moving object in the following manner: detects an image area that is moving in the image corresponding to the image signals supplied from the surveillance camera; tracks the detected image area for a predetermined period of time; and then determines whether or not the image area corresponds to a moving object based on the tracking result (see Jpn. Pat. Appln. Laid-Open Publication No. 2000-322581 and Non-Patent Document “Local correlation calculation for detecting and tracking movements” by Toshihiko Morita, Institute of Electronics, Information and Communication Engineers (IEICE) transactions, D-II, VOL. J84-D-II, No. 2, pp. 299-309, February 2001, for example).
In addition, another moving object detection device is proposed which can detect a moving object in the following manner: detects change in the image corresponding to the image signals supplied from the surveillance camera; sets the area where the change occurred as a changing area; disposes a plurality of correlation calculation blocks in the changing area to perform correlation calculation; and then detects a moving object based on results of the correlation calculation (see Jpn. Pat. Appln. Laid-Open Publication No. 2003-169319, for example).
FIG. 39 shows the configuration of the moving object detection device 1 according to said Non-patent document by Toshihiko Morita.
The moving object detection device 1 receives image data D1 from surveillance cameras or the like, and then supplies the image data D1 via an image input section 2 to a detection block setting section 3. The detection block setting section 3 divides an image corresponding to the image data D1 into a plurality of detection blocks (areas) BR1 as shown in FIG. 40A. In this case, the blocks BR1 are surrounded by dotted lines in FIG. 40A. The detection block setting section 3 then supplies image data of each detection block BR1 to a correlation calculation block disposition section 4 as detection block data D2.
In this case, the first division method is stationary. Therefore, initial position and size of each detection block BR1 always become constant. However, when the detection block BR1 goes into tracking state as described below, this detection block BR1 sequentially moves like FIG. 40B in dependence upon vector information D8 about a representative motion vector. This vector information D8 is based on the motion vectors of detection block BR1 supplied from a subsequent section (i.e., a spatial motion uniformity determination section 9).
The correlation calculation block disposition section 4 performs division process for each detection block data D2 supplied from the detection block setting section 3 to divide each detection block BR1 into a plurality of correlation calculation blocks BR2. Here, the correlation calculation blocks BR2 are indicated by ruled lines in FIG. 40A. The correlation calculation block disposition section 4 then supplies image data of each correlation calculation block BR2 to a correlation calculation execution section 6 as correlation calculation block data D3. When the process of the previous section (i.e., the detection block setting section 3) moves the detection block BR1, the correlation calculation blocks BR2 in the detection block BR1 moves accordingly.
The image data D1 input through the image input section 2 is also supplied to a previous image storage section 5. The previous image storage section 5 stores the image data D1 accordingly. After that, at the timing when the correlation calculation block disposition section 4 supplies the correlation calculation block data D3 to the correlation calculation execution section 6, an image piece which corresponds to the correlation calculation block BR2 in terms of both position and size is extracted from the preceding frame of image data D1 stored in the previous image storage section 5 and is read out by the correlation calculation execution section 6 as delayed image data D4.
The correlation calculation execution section 6 performs correlation calculation for the correlation calculation block data D3 supplied from the correlation calculation block disposition section 4 and the delayed image data D4 read from the previous image storage section 5. That is to say, the correlation calculation execution section 6 calculates correlation between the current image of each correlation calculation block BR2 and the corresponding previous image. And then the correlation calculation execution section 6 generates a correlation value map of each correlation calculation block BR2 based on the calculation result. The correlation calculation execution section 6 subsequently recognizes a vector on the correlation value map which is pointing out from an original point to a minimum value point as a motion vector of the correlation calculation block BR2.
In the case in which the minimum value on the correlation value map shows a distinct negative peak, the motion vector is deemed to have high reliability; and otherwise, the motion vector is deemed to have low reliability. Therefore, in the case in which: the magnitude of the motion vector is not equal to “0”; and the minimum value on the correlation value map shows a distinct negative peak, the correlation calculation execution section 6 determines that the correlation calculation block BR2 has “movements”, and otherwise determines that the correlation calculation block BR2 has “no movements”. The correlation calculation execution section 6 then supplies correlation calculation result information D5 showing the determination result to an initial detection section 7.
The initial detection section 7 performs counting process for each detection block BR1, based on the correlation calculation result information D5 supplied. That is to say, the initial detection section 7 counts the number of the correlation calculation block BR2 that was determined to have “movements” for each detection block BR1. When the counted number is greater or equal to a predetermined number, the initial detection section 7 determines that the detection block BR1 has “movements”. The initial detection section 7 then supplies initial detection determination information D6 showing the determination result to a spatial motion vector integration section 8. By the way, the detection block BR1 that was determined to have “movements” will go into the tracking state (FIG. 40B).
Based on the initial detection determination information D6, the spatial motion vector integration section 8 performs accumulation process for each detection block BR1 which was determined to have “movements”. In the accumulation process, the spatial motion vector integration section 8 accumulates the correlation value maps of all the correlation calculation blocks BR2 in the detection block BR1. The spatial motion vector integration section 8 then supplies spatial motion vector integration information D7 showing the accumulation results to the spatial motion uniformity determination section 9.
Based on the spatial motion vector integration information D7 supplied, the spatial motion uniformity determination section 9 processes each detection block BR1 which was determined to have “movements” by the initial detection section 7 in the following manner; finds out a vector on the accumulated correlation value map which is pointing out from an original point to a minimum value point as a motion vector (the vector is shown by arrows in FIG. 40A); and then recognizes the motion vector as a representative motion vector of the detection block BR1 when the motion vector is not equal to “0” and the minimum value of accumulated correlation value maps shows a distinct negative peak.
The spatial motion uniformity determination section 9 then supplies vector information D8 about the representative motion vector to the detection block setting section 3. Based on the vector information D8, the detection block setting section 3 allows each detection block BR1 which was determined to have “movements” by the initial detection section 7 to go into the tracking state. In the tracking state, the detection blocks BR1 move in accordance with the magnitude and direction of the representative motion vector which is based on the vector information D8, as shown in FIG. 40B. That allows detecting “movements” of each detection block BR1 that was determined to have “movements”. The tracking process of tracking the detection blocks having “movements” is repeated until a predetermined period of tracking time has passed.
In addition, the spatial motion uniformity determination section 9 supplies map information D9 to a temporal motion vector integration section 10. The map information D9 shows information about the spatially-accumulated correlation value maps in each detection block BR1 being tracked.
Based on the map information D9 supplied, the temporal motion vector integration section 10 continues to temporally accumulate the spatially-accumulated correlation value maps of each detection block BR1 until the predetermined period of tracking time has passed. Therefore, the spatially and temporally accumulated correlation value map of each detection block BR1 is generated when the period of tracking time has ended. In this manner, the temporal motion vector integration section 10 obtains tracking information showing the spatially and temporally accumulated correlation value map of each detection block BR1. The temporal motion vector integration section 10 then supplies map information D10 to a temporal motion uniformity determination section 11. The map information D10 shows information about the spatially and temporally accumulated correlation value map of each detection block BR1.
Based on the map information D10 supplied, the temporal motion uniformity determination section 11 performs processes in the following manner: on the correlation value map obtained by completing the spatial and temporal accumulation, recognizes a vector which is pointing out from an original point to a minimum value point as a motion vector; and determines that the corresponding detection block BR1 has “movements” when the motion vector is not equal to “0” and the minimum value of the correlation value map shows a distinct negative peak.
The following is the reason: in general, moving objects such as human beings and vehicles moves in a fixed direction in short term, therefore, when detecting the movements of those kinds of moving object, the spatially and temporally accumulated correlation value map has a negative peak of minimum value at almost the same position every time. By contrast, background movements such as natural trees and waves are vibrating, therefore, when detecting the movements of trees, waves or the like, the spatially accumulated correlation value map has a negative peak of minimum value at different positions every time.
In this manner, the moving object detection device 1 can accurately detect moving objects such as human beings and vehicles, based on the “movements” of objects moving in the image corresponding to the inputted image data D1.
However, in the case in which: there is a plurality of moving objects and background movements in one detection block BR1; and they are moving in different directions, the motion vectors cancel each other. In this case, the above-noted moving object detection device 1 may fail to detect the moving objects.
To solve this problem, there is proposed moving object detection method in Jpn. Pat. Appln. Laid-Open Publication No. 2003-169319. FIG. 41 (the parts of FIG. 41 have been designated by the same symbols and marks as the corresponding parts of FIG. 39) shows a moving object detection device 20 according to the moving object detection method of Jpn. Pat. Appln. Laid-Open Publication No. 2003-169319. The configuration of the moving object detection device 20 is almost similar to that of the above-noted moving object detection device 1 (FIG. 39) except for an image change extraction section 21 replacing the detection block setting section 3 (FIG. 39) and initial detection section 7 (FIG. 39).
In this case, as shown in FIG. 42, the image change extraction section 21 includes: a background difference section 30 for removing background images from the image corresponding to the image data D1 supplied from an image input section 2; a background update section 31 for updating the background images; and a changing area extraction section 32 for setting a changing area in the image whose background images have been removed by the background difference section 30.
The background difference section 30 calculates difference between the image corresponding to the image data D1 and an image corresponding to background image data D20 supplied from the background update section 31 (the background image data D20 has been saved in the background update section 31). The background difference section 30 then removes the image corresponding to the background image data from the image corresponding to the image data D1 to extract an image where a change occurs. The background difference section 30 subsequently transmits the extracted image to the background update section 31 and the changing area extraction section 32 as extraction, image data D21.
The background update section 31 for example averages a predetermined number of the images corresponding to the image data D1 to obtain average image. In the average image, outlines of moving parts have been overlapped with each other, and therefore their appearances are indistinct. The background update section 31 appropriately performs update process to remove indistinct objects from the average images, and stores the images as the background image data D20.
On the image whose background image has been removed by the background difference section 30, the changing area extraction section 32 sets a changing area AR to the part where change occurs as shown in FIG. 43A, and then generates changing area setting information D11 showing the result. In FIG. 43A, the changing area AR is surrounded by dotted lines. The changing area extraction section 32 subsequently supplies the changing area setting information D11 to the correlation calculation block disposition section 22.
The correlation calculation block disposition section 22 performs division process for each changing area AR supplied from the image change extraction section 21. In the division process, the correlation calculation block disposition section 22 divides the changing area AR into a plurality of correlation calculation blocks BR2 (FIG. 43A), and then supplies image data of each correlation calculation block BR2 to a correlation calculation execution section 23 as correlation calculation block data D12. When the changing area AR moves as a result of the process of the image change extraction section 21, each correlation calculation block BR2 in the changing area AR also moves accordingly.
Based on the correlation calculation block data D12 supplied from the correlation calculation block disposition section 22 and delayed image data D4 read from the previous image storage section 5, the correlation calculation execution section 23 performs correlation calculation for each correlation calculation block BR2. In this correlation calculation, the correlation calculation execution section 23 calculates correlation between the current image of each correlation calculation block BR2 and the corresponding previous image. And then the correlation calculation execution section 23 generates a correlation value map of each correlation calculation block BR2 based on the calculation result. The correlation calculation execution section 23 subsequently recognizes a vector on the correlation value map which is pointing out from an original point to a minimum value point as a motion vector of the correlation calculation block BR2.
In the case in which: the motion vector of the correlation calculation block BR2 is not equal to “0”; and the minimum value in the correlation value map shows a distinct negative peak, the correlation calculation execution section 23 determines that the correlation calculation block BR2 has “movements”; and otherwise determines that the correlation calculation block BR2 has “no movements”. The correlation calculation execution section 23 then supplies correlation calculation result information D13 showing the determination result to a spatial motion vector integration section 24.
Based on the correlation calculation result information D13, the spatial motion vector integration section 24 performs accumulation process for each changing area AR. In this accumulation process, the spatial motion vector integration section 24 accumulates the correlation value maps of all the correlation calculation blocks BR2 in the changing area AR. The spatial motion vector integration section 24 then supplies spatial motion vector integration information D14 showing the accumulation results to the spatial motion uniformity determination section 25.
Based on the spatial motion vector integration information D14 supplied, the spatial motion uniformity determination section 25 processes each changing area AR in the following manner: finds out a vector on the accumulated correlation value map which is pointing out from an original point to a minimum value point as a motion vector (the vector is shown by arrows in FIG. 43A; and then recognizes the motion vector as a representative motion vector of the changing area AR when the motion vector is not equal to “0” and the minimum value of accumulated correlation value maps shows a distinct negative peak.
The spatial motion uniformity determination section 25 then supplies vector information D15 about the representative motion vector to the correlation calculation block disposition section 22. Based on the vector information D15, the correlation calculation block disposition section 22 allows each changing area AR to go into the tracking state. As a result, the correlation calculation block disposition section 22 starts to supply correlation calculation block data relevant to the changing area AR to the correlation calculation execution section 23. That allows detecting the “movements” of each changing area AR. This tracking process is repeated until a predetermined period of tracking time has passed.
In addition, correlation value maps are spatially accumulated in each changing area AR being tracked. The spatial motion uniformity determination section 25 supplies map information D9 about the spatially-accumulated correlation value maps to the temporal motion vector integration section 10.
After that, the temporal motion vector integration section 10 and the temporal motion uniformity determination section 11 performs processes in the same way as the above-noted moving object detection device 1: the temporal motion vector integration section 10 spatially and temporally accumulates the correlation value maps of the changing area AR that is deemed to be moving objects for the predetermined period of tracking time; and then the temporal motion uniformity determination section 11 detects the changing area AR having “movements”.
In this manner, the moving object detection device 20 can accurately detect moving objects such as human beings and vehicles, based on the “movements” of objects moving in the image corresponding to the inputted image data D1.
By the way, the above-noted moving object detection device 1, 20 first set an area to detect movements. And then the moving object detection device 1, 20 spatially and temporally evaluate the uniformity of movements while tracking the area for a certain period of time. Based on the evaluation, the moving object detection device 1, 20 determine whether or not the area corresponds to moving objects.
However, the area set by the above method may include not only moving objects but also other things.
The following is description of the moving object detection device 1. For example as shown in FIG. 44, when a person HU is walking through in front of a background BG (such as advertising flags on the side of roads) that is moving, the detection block BR1 includes both the motion vectors corresponding to the person HU and the motion vectors corresponding to the movements of the background BG. In this case, the movements of person HU do not correlate with the background movements. Therefore, the moving object detection device 1 does not determine that the detection block BR1 has “movements”, since the spatially-integrated correlation value maps of the detection block BR1 does not show any distinct peaks. Even if the moving object detection device 1 determines that the detection block BR1 has “movements”, the representative motion vector does not represent the movements of person HU correctly. Therefore, the moving object detection device 1 may fail to track and detect the person HU.
The following is description of the moving object detection device 20. When the person HU who is moving is very close to the background BG that is moving, the moving object detection device 20 can correctly detect the person HU, because it has previously set the changing area AR (FIGS. 43A and 43B) based on background difference. However, as shown in FIG. 45, when the background BG that is moving on a screen and the person HU overlap each other, the changing area AR includes both the motion vectors corresponding to the person HU and the motion vectors corresponding to the movements of the background BG. Under this condition, the moving object detection device 20 may fail to track and detect the person HU like the moving object detection device 1.
When the moving object detection device 20 has been desensitized toward background difference, the moving object detection device 20 does not detect the movements of background BG as the changing area AR. However, in the case in which the brightness level of the movements of person HU is close to that of the background movements, the moving object detection device 20 does not detect the movements of person HU as the changing area AR. Accordingly, the moving object detection device 20 may fail to track and detect the person HU like the moving object detection device 1, even if it has adjusted the sensitivity for background difference.
In general, a video surveillance system equipped with those kinds of moving object detection device 1, 20 are often installed in the environment where illumination is not sufficiently bright, and therefore the SN ratio of the images corresponding to the image signals input from surveillance cameras are not good. In this case, random noise is not negligible against the contrast of subjects. The random noise is generated all over the screen, and varies with time. That causes problems similar to those arising from the overlapping of the background change and the moving object as described above.
In addition, the moving object detection device 1, 20 described above first sets the area (the detection block area BR1 or the changing area AR), and moves it in conjunction with tracking. The region from which the area has left is not covered with any correlation calculation block BR2. Accordingly, the moving object detection device 1, 20 would not detect a moving object coming into the region.
That is to say, as shown in FIGS. 46A and 46B, when a second person HU2 who is following a first person HU1 appears on the screen while the moving object detection device 1 is tracking the first person HU by moving the detection block BR1 (FIG. 46A), the detection block BR1 tracking the first person HU1 has already left its initial position to track the first person HU1, and therefore the initial position is not covered with any correlation calculation blocks. Accordingly, the moving object detection device 1 does not detect the second person HU2 (FIG. 46B).
The following is description of the moving object detection device 20. For example in FIGS. 47A and 47B, while the moving object detection device 20 is tracking a first person HU1 by moving the correlation calculation blocks BR2 after having set the changing area AR for the first person HU1 on the screen (FIG. 47A), a second person HU2 who is following the first person HU1 appears on the screen. Even if the moving object detection device 20 sets a new changing area AR that surrounds both the first person HU1 and the second person HU2 (FIG. 47B), the second person HU2 is not covered with the correlation calculation blocks BR2, because the moving object detection device 20 does not change the positions of correlation calculation blocks BR2 within the changing area AR whose shape has changed in this tracking period. Therefore, the moving object detection device 20 does not detect the movements of second person HU2.
Even if the moving object detection device 20 moves the correlation calculation blocks BR2 in conjunction with the tracking, that increases the complexity of the image processing procedure and device configuration, because it has to temporally change the positions of the correlation calculation blocks BR2 in each area independently.
Thus, the above-noted moving object detection device 1, 20 have various kinds of problems such as they have difficulty in accurately tracking and detecting moving objects, because they first set the area (the detection block BR1 or the changing area AR) to the target prior to evaluating the uniformity of “movements” and move the correlation calculation blocks BR2.